Image processing device, image processing method, and image processing program

ABSTRACT

An image processing device to improve measurement accuracy for a measurement point includes a corresponding point adjusting unit that causes a display unit to display a measurement point peripheral image that is an image of an area in a periphery of the measurement point extracted from a reference image and a corresponding point peripheral image that is an image of an area in a periphery of a corresponding point extracted from a corresponding image, the measurement point peripheral image and the corresponding point peripheral image being displayed side by side, to adjust a position of the corresponding point found by a corresponding point searching unit.

TECHNICAL FIELD

One aspect of the present invention relates to a technique for measuringthree-dimensional coordinates of a measurement point from a referenceimage and a corresponding image.

BACKGROUND ART

A known conventional method (stereoscopic measurement) of measuring alength, a three-dimensional position (coordinates), or the like of asubject involves using two cameras (stereo cameras). The stereoscopicmeasurement is performed as follows. Specifically, images of a subjectare captured with two cameras arranged at different positions, and thethree-dimensional position of the subject is calculated based on aparallax between the subjects in two images captured by the two cameras(difference between coordinates of a measurement point on one of theimages and coordinates of a corresponding point on the other one of theimages corresponding to the measurement point). Then, the length of thesubject, the distance to the subject, and the like are measured.Measurement accuracy of the stereoscopic measurement varies depending ona distance to a subject, a baseline length that is a distance betweentwo cameras, a focal length of a lens, a specification of the camerassuch as the number of pixels, parallax calculation accuracy, positiondesignation accuracy of a measurement target on an image, and the like.

An example of a known technique for calculating a parallax includesblock matching in which similarity between images is evaluated. Blockmatching is a method including searching one of images for an area withthe highest similarity with respect to an area selected in the other oneof the images and calculating a parallax from a difference between theimages in the position (coordinates) of an evaluation target area.Unfortunately, the method is affected by noise in the image, the angleof the subject, the pattern of the subject, and the like, and thus doesnot necessarily result in a correct parallax calculated. In thiscontext, Patent Literature 1 discloses a method of improving theparallax calculation accuracy. Specifically, the technique disclosed inPatent Literature 1 enables a user to adjust the coordinates of acorresponding point by operating a designator while checking acorresponding image in which a corresponding area including thecorresponding point is displayed, and moving the corresponding area sothat the corresponding area corresponds to a reference area on thereference image.

CITATION LIST Patent Literature

PTL1: JP 2002-5659 A (published on Jan. 9, 2002)

SUMMARY OF INVENTION Technical Problem

The method disclosed in Patent Literature 1 has the fall wing problems.In Patent Literature 1, a display unit displays a reference imageincluding a mark indicating the reference area and a corresponding imageincluding a mark indicating the corresponding area. The user searchesfor the corresponding area corresponding to the reference area by movingthe mark indicating the correspond area. With the technique, it isdifficult for the user to adjust the corresponding point to a correctposition by moving the corresponding area while visually checking theimage, if a subject has depth and thus the reference area and thecorresponding area include portions with a large difference in parallax.This is because adjustment on the position of the corresponding area onthe corresponding image performed so that a part of the correspondingarea matches a corresponding part of the reference area results indisplacement between other portions. Furthermore, it is difficult forthe user to adjust the position of the corresponding point in detailwhile visually checking the corresponding image, depending on thedisplay size (magnitude) of the corresponding image, that is, in a casewhere the image is displayed with a reduced size or the like.

An aspect of the present invention is made in view of the aboveproblems, and an object of the aspect is to provide a method with whicha user measuring a measuring point can perform accurate adjustment sothat a position of a corresponding point on a corresponding imagecorresponds to the measurement point on a reference image, to improvemeasurement accuracy for the measurement point.

Solution to Problem

An image processing device according to one aspect of the presentinvention for solving the problems described above includes acorresponding point searching unit configured to search a correspondingimage for a corresponding point corresponding to a measurement point Ona reference image obtained by capturing an image of a subject, thecorresponding image being obtained by capturing an image of the subjectfrom a point of view different from a point of view for the referenceimage, a corresponding point adjusting unit that causes a display unitto display a measurement point peripheral image that is an image of anarea in a periphery of the measurement point extracted from thereference image and a corresponding point peripheral image that is animage of an area in a periphery of the corresponding point extractedfrom the corresponding image, the measurement point peripheral image andthe corresponding point peripheral image being displayed side by side,and is configured to adjust a position of the corresponding point basedon an instruction input to an input unit, and a calculating unitconfigured to calculate three-dimensional coordinates of the measurementpoint on the subject based on a position of the measurement point on thereference image and the position of the corresponding point on thecorresponding image.

An image processing method according to one aspect of the presentinvention includes searching a corresponding image for a correspondingpoint corresponding to a measurement point on a reference image obtainedby capturing an image of a subject, the corresponding image beingobtained by capturing an image of the subject from a point of viewdifferent from a point of view for the reference image,

causing a display unit to display a measurement point peripheral imagethat is an image of an area in a periphery of the measurement pointextracted from the reference image and a corresponding point peripheralimage that is an image of an area in a periphery of the correspondingpoint extracted from the corresponding image, the measurement pointperipheral image and the corresponding point peripheral image beingdisplayed side by side, and adjusting a position of the correspondingpoint based on an instruction input to an input unit, and

calculating three-dimensional coordinates of the measurement point onthe subject based on a position of the measurement point on thereference image and the position of the corresponding point on thecorresponding image.

Advantageous Effects of invention

One aspect of the present invention enables a user measuring a measuringpoint to perform accurate adjustment so that a position of acorresponding point on a corresponding image corresponds to themeasurement point on a reference image, so that measurement accuracy canbe improved for the measurement point.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example of a configuration ofan image processing device according to a first embodiment.

FIG. 2 is a diagram illustrating a mechanism of stereoscopic measurementperformed with an image capturer.

FIG. 3 is a flowchart illustrating an image processing method accordingto Aspect 1 of a first embodiment of the present invention.

FIG. 4 is a flowchart illustrating an image processing method accordingto Aspect 2 of a first embodiment of the present invention.

FIG. 5 is a flowchart illustrating an image processing method accordingto Aspect 3 of a first embodiment of the present invention.

FIG. 6 is a flowchart illustrating an image processing method accordingto Aspect 4 of a first embodiment of the present invention.

FIG. 7A is a diagram illustrating a reference image displayed by adisplay unit. FIG. 7B is a diagram illustrating a measurement pointperipheral image and a corresponding point peripheral image displayed bya display unit.

FIG. 8 is a diagram illustrating a display unit displaying a referenceimage, a measurement point peripheral image, and a corresponding pointperipheral image.

FIG. 9A and FIG. 9B are diagrams illustrating a measurement pointperipheral image and a corresponding point peripheral image displayed bya display unit.

FIG. 10A and FIG. 10B are diagrams illustrating a measurement pointperipheral image and a corresponding point peripheral image displayed bya display unit.

FIG. 11 is a diagram illustrating a display unit displaying a referenceimage, a measurement point peripheral image, and a corresponding pointperipheral image.

FIG. 12 is a diagram illustrating a display unit displaying a referenceimage, a measurement point peripheral image, and a corresponding pointperipheral image.

FIG. 13 is a diagram illustrating a display unit displaying a referenceimage, a measurement point peripheral image, and a corresponding pointperipheral image.

FIG. 14 is a diagram illustrating a display unit displaying a referenceimage, a measurement point peripheral, image, and a corresponding pointperipheral image.

FIG. 15A is a diagram illustrating a reference image and a correspondingimage displayed by a display unit. FIG. 15B is a diagram illustrating ameasurement point peripheral image and a corresponding point peripheralimage displayed by a display unit.

FIG. 16 is a diagram illustrating a measurement point peripheral imageand a corresponding point peripheral image displayed by a display unit.

DESCRIPTION OF EMBODIMENTS First Embodiment

Hereinafter, one embodiment of the present invention is described indetail with reference to the drawings.

Image Processing Device

FIG. 1 is a block diagram illustrating a configuration of an imageprocessing device 1 according to the present embodiment. As illustratedin FIG. 1, the image processing device 1 includes an image capturer 2, astorage 3, an input unit 4, a measurer 5 (calculating unit), and adisplay unit 11. In the description below, an image referred to by auser for designating a measurement point and an image used by the userfor searching for a corresponding point corresponding to themeasurement, in stereoscopic measurement, are respectively referred toas a reference image and a corresponding image.

The image capturer 2 includes a first image capturing device 12 and asecond image capturing device 13 (illustrated in FIG. 2 described later)and acquires a reference image and a corresponding image correspondingto the reference image, with these image capturing devices capturing animage of a common subject. An example of the image capturer includes amember that includes an, optical system such as a lens module, an imagesensor such as a Charge Coupled Device (CCD) and a Complementary MetalOxide Semiconductor (CMOS), an analog signal processing unit, and anAnalog/Digital (A/D) converting unit, and that outputs a signal from theimage sensor as an image.

In an example or the present embodiment described below, thecorresponding image referred to by the image processing device 1 is asingle image captured by either the first image capturing device 12 andthe second image capturing device 13 of the image capturer 2.Alternatively, the corresponding image may include a plurality of imagescaptured by a plurality of image capturing devices. A method similar toan image processing method according to the present embodiment may beapplied to such a configuration. The image processing device 1 accordingto the present embodiment includes the image capturer 2. Note that thescope of an aspect of the present invention includes an aspect withoutthe image capturer 2. In such a case, reference and corresponding imagescaptured by a plurality of image capturing devices in advance are usedfor measuring a measurement point.

The storage 3 stores therein camera parameters such as a baseline lengthand a focal length required for measuring a measurement point. Thestorage 3 according to the present embodiment stores therein a cameraparameter of the image capturer 2. In a case where the camera parameterchanges due to a change in the arrangement of each of the first imagecapturing device 12 and the second image capturing device 13 of theimage capturer 2, the user may update the camera parameter by using theinput unit 4 and the storage 3 may receive the new camera parameter fromthe image capturer 2.

The input unit 4 receives a measurement point, in a subject on thereference image, input from the user (operator). Examples of the inputunit 4 include an input device such as a mouse and a keyboard.

The measurer 5 is a member that performs each step in an imageprocessing method according to one aspect of the present invention, andmay be a semiconductor circuit such as a central processing unit (CPU)or a large-scale integration (LSI). As illustrated in FIG. 1, themeasurer 5 includes an receiver 6, a measurement point receiver 7, acorresponding point searching, unit 8; a corresponding point adjustingunit 9, and a measurement value calculating unit 10. The image receiver6 receives the reference image and the corresponding image captured bythe image capturer 2. The measurement point receiver 7 receives ameasurement point (a point on the reference image corresponding to themeasurement point) input on the reference image by the user through theinput unit 4. The corresponding point searching unit 8 searches for andsets a position of the corresponding point corresponding to themeasurement point on the reference image received by the measurementpoint receiver 7 by using a conventionally known method such as blockmatching. The corresponding point adjusting unit 9 receives a positionof a corresponding point as a result of adjustment by the user throughthe input unit 4 performed so that the corresponding point, set by thecorresponding point searching unit 8, further corresponds to a positionof the measurement point on a corresponding point peripheral image. Themeasurement value calculating unit 10 refers to the camera parameterstored in the storage 3, the position of the measurement point on thereference image received by the measurement point receiver 7. and theposition of the corresponding point on the corresponding image receivedby the corresponding point adjusting unit 9, to calculate thethree-dimensional coordinates of the measurement point. The measurementvalue calculating unit 10 calculates a distance to a subject that is ameasurement target, a length of the subject, or the like, from thethree-dimensional coordinates of the measurement point thus calculated.

The display unit 11 displays an image (measurement point peripheralimage) of an area in the periphery of the measurement point on thereference image and an image (corresponding point peripheral image) ofan area in the periphery of the corresponding point on the correspondingimage, extracted from the reference image and the corresponding imagereceived by the image receiver 6. The images are displayed side by side.The display unit 11 displays the positions of the measurement points andthe corresponding points as a result of processes, performed by thecomponents of the measurer 5, on the measurement point peripheral imageand the corresponding point peripheral image, as well as a progress ofthe processes. The display unit 11 further displays the position of themeasurement point calculated by the measurement value calculating unit10, information about the subject calculated from the position, andinformation about the image capturer 2 stored in the storage 3.

The display unit 11 may be a conventionally known display device such asa liquid crystal display for example. The input unit 4 and the displayunit 11 described above may be integrated. An example of the input unit4 and the display unit 11 integrated includes a touch panel. The touchpanel is a device that senses a touch of the user on a picture or anarea displayed on a display screen, and outputs a signal, indicatinginformation about the sensing, to the outside. An example of the touchpanel includes a resistive touch panel that senses the voltage at theposition touched by the user and a capacitive touch panel that detectsthe position by sensing a change in static capacitance between afingertip and a conducting layer. The device with a touch panel canperform an operation corresponding to information about the position ofor an operation performed with a finger of the user on the screen.

<Image Processing Method> (Measurement Error in StereoscopicMeasurement)

Before an image processing method according to the present embodiment isdescribed, first of all, a measurement error in stereoscopic measurementwill be described with reference to FIG. 2. FIG. 2 is a diagramillustrating a mechanism of the stereoscopic measurement. In FIG. 2, thefirst image capturing device 12 is arranged on the left side and thesecond image capturing device 13 is arranged on the right side. Each ofthe devices has an optical axis extending along a Z axis direction inFIG. 2. The first image capturing device 12 includes an image sensor 14and a lens 15. The second image capturing device 13 includes an imagesensor 16 and a lens 17. The baseline length and the focal length ofeach of the first image capturing device 12 and the second imagecapturing device 13 are respectively denoted with B and In FIG. 2, whena coordinate system of the first image capturing device 12 serving as areference image capturing device for the measurement is used, thethree-dimensional coordinates (X, Y, Z) of the measurement point arecalculated with the following Formula (1), Formula (2), and Formula (3),by using the parallax d (a difference between the coordinates of themeasurement point on an image captured by the first image capturingdevice 12 and the coordinates of the corresponding point on an imagecaptured by the second image capturing device 13), the focal length f,and the baseline length B.

Z=B×f/d   (1)

X=x×Z/f   (2)

Y=y×Z/f   (3)

Here, x in Formula (2) and y in Formula (3) respectively represent an xcoordinate and a y coordinate of a measurement point K on the imagecaptured by the first image capturing device 12. As is apparent fromFormula (1), the parallax d needs to be correctly calculated tocalculate a Z coordinate in the three-dimensional coordinates of themeasurement point with the stereoscopic measurement. Each of Formula (2)and Formula (3) has the right side including Z. Thus, in a case wherethe parallax d fails to be correctly calculated and thus results in theZ coordinate calculated with Formula (1) being different from the actualcoordinate, the X coordinate and the Y coordinate are also shifted.Thus, the calculation accuracy for the parallax d affects themeasurement accuracy of the stereoscopic measurement. In view of theabove, a method for improving the measurement accuracy according to thepresent invention is described below. The method is achieved with theparallax correctly calculated with the corresponding point adjusted bythe user to he at the position corresponding to the measurement point Keven in a case where there is an error in the parallax with thecorresponding point found by the corresponding point searching unit 8not corresponding to the measurement point K.

(Image Processing Method According to Aspect 1)

FIG. 3 is a flowchart illustrating an image processing method accordingto Aspect 1 of the present embodiment performed by the image processingdevice 1. The display unit 11 displays the reference image and thecorresponding image received by the image receiver as well as themeasurement point peripheral image and the corresponding pointperipheral image through steps S0 to S4 described later. The userperforms operations, required for measuring a measurement point asdescribed later, through the input unit 4 by referring to the imagesdisplayed by the display unit 11. The measurement point peripheral imageand the corresponding point peripheral image are described more indetail in a later section (corresponding point adjustment by user).

The image receiver 6 receives the reference image and the correspondingimage respectively acquired by the first image capturing device 12 andthe second image capturing device 13 (step S0). Any one of the twoimages captured by the first image capturing device 12 and the secondimage capturing device 13 may be selected as the reference image by theuser or by the image receiver 6 automatically. The display unit 11displays the reference image and the corresponding image received by theimage receiver 6 in response to step S0. Further in response to step S0,the display unit 1 extracts an image of an area in the periphery of themeasurement point (measurement point peripheral image) in the referenceimage and an image of an area in the periphery of the correspondingpoint (corresponding point peripheral image) in the corresponding image,respectively from the reference image and the corresponding imagereceived by the image receiver 6, and displays the images side by side.

Next, the measurement point receiver 7 receives the measurement pointinput by the user on the reference image through the input unit 4 (stepS1). For example, the user can select a point at a desired position, onthe reference image displayed by the display unit 11, as the coordinatesof the measurement point by operating a mouse or a four-way controllerto move a cursor on the reference image or to perform the other likeoperation. The user may further input a numerical value indicating thecoordinates of the measurement point through the input unit 4 todesignate the position of the measurement point on the reference image.The configuration where the user inputs the numerical value indicatingthe coordinates of the measurement point, enables the user to check theposition of the measurement point on the reference image, with a pointeror the like indicating the position of the coordinates input by the userdisplayed on the reference image displayed by the display unit 11 in anoverlapping manner.

Next, the corresponding point searching unit 8 searches for and sets thecorresponding point corresponding to the measurement point received bythe measurement point receiver 7, on the corresponding image (step S2).For example, the corresponding point searching unit 8 can use aconventionally known method such as block matching to search for thecorresponding point.

Next, the user adjusts the corresponding point set by the correspondingpoint searching unit 8 through the input unit 4 while referring to themeasurement point peripheral image and the corresponding pointperipheral image described above displayed on the display unit 11, sothat the corresponding point further corresponds to a position of themeasurement point on the corresponding point peripheral image(corresponding point adjustment step). Then, the corresponding pointadjusting unit 9 receives the position of the corresponding point thusadjusted by the user (step S3). How the user adjusts the position of thecorresponding point through the input unit 4 while referring to themeasurement point peripheral image and the corresponding pointperipheral image displayed on the display unit 11 in this step isdescribed more in detail in the later described section (correspondingpoint adjustment by user).

Next, the measurement value calculating unit 10 calculates thethree-dimensional coordinates of the measurement point by referring toand based on the camera parameter stored in the storage 3, the positionof the measurement point on the reference image received by themeasurement point receiver 7, and the position of the correspondingpoint on the corresponding image received by the corresponding pointadjusting unit 9 (step S4, calculation step). The measurement valuecalculating unit 10 may calculate the distance to the subject that isthe measurement target, the length of the subject, or the like, from thethree-dimensional coordinates of the measurement point thus calculated.The three-dimensional coordinates of the measurement point calculated bythe measurement value calculating unit 10 are not particularly limitedand may be coordinates obtained with the first image capturing device 12or the second image capturing device 13 that has captured the referenceimage or the corresponding image, or may be any coordinates set inadvance. In response to step S4, the display unit 11 may display thethree-dimensional coordinates of the measurement point calculated by themeasurement value calculating unit 10 and each measurement value relatedto the subject calculated from the coordinates.

(Image Processing Method According to Aspect 2)

FIG. 4 is a flowchart illustrating, an image processing method accordingto Aspect 2 of the present embodiment performed by the image processingdevice 1. The image processing method according to the present aspect isthe same as the image processing method according to Aspect 1, exceptthat the order of steps S0 to S4 performed by the image processingdevice 1 is changed and that step S4′ with a process that issubstantially similar to that of step S4 is performed. Thus, detaildescription of steps denoted with the same reference number as thecounterparts in Aspect 1 will he omitted.

As illustrated in FIG. 4, the image receiver 6 receives the referenceimage and the corresponding image respectively acquired by the firstimage capturing device 12 and the second image capturing device 13 (stepS0). Next, the measurement point receiver 7 receives the measurementpoint input by the user on the reference image through the input unit 4(step S1). Next, the corresponding point searching unit 8 searches forand sets the corresponding point corresponding to the measurement pointreceived by the measurement point receiver 7, on the corresponding image(step S2, corresponding point search step).

Next, the measurement value calculating unit 10 calculates thethree-dimensional coordinates of the measurement point by referring, toand based on the camera parameter stored in the storage 3, the positionof the measurement point on the reference image received by themeasurement point receiver 7, and, the position of the correspondingpoint on the corresponding image set by the corresponding pointsearching unit 8 (step S4′, calculation step).

Next, the user adjusts the corresponding point set by the correspondingpoint searching unit 8 through the input unit 4 while referring to themeasurement point peripheral image and the corresponding pointperipheral image described above displayed on the display unit, so thatthe corresponding point further corresponds to a position of themeasurement point on the corresponding point peripheral image(corresponding point adjustment step). Then, the corresponding pointadjusting unit 9 receives the position of the corresponding point thusadjusted by the user (step S3).

Next, the measurement Value calculating unit 10 calculates thethree-dimensional coordinates of the measurement point by referring toand based on the camera parameter stored in the storage 3, the positionof the measurement point on the reference image received by themeasurement point receiver 7, and the position of the correspondingpoint on the corresponding image received by the corresponding pointadjusting unit 9 (step S4).

The display unit 11 displays the three-dimensional coordinates of themeasurement point calculated by the measurement value calculating unit10 in step S4′ and the three-dimensional coordinates of the measurementpoint calculated by the measurement value calculating unit 10 in stepS4.

With the image processing method according to Aspect 2 described above,the user can compare the three-dimensional coordinates of themeasurement point calculated by the measurement value calculating unit10 in step S4′ with the three-dimensional coordinates of the measurementpoint calculated by the measurement value calculating unit 10 in stepS4. Thus, the user can check whether the corresponding point searchingunit 8 has accurately set the corresponding point corresponding to themeasurement point.

(Image Processing Method According to Aspect 3)

FIG. 5 is a flowchart illustrating an image processing method accordingto Aspect 3 of the present embodiment performed by the image processingdevice 1. The image processing method according to the present aspect isthe same as the image processing method according to Aspect 1 exceptthat step S4′ is added and step S5 is further added. Thus, detaildescription of steps denoted with the same reference number as thecounterparts in Aspect 1 will be omitted.

As illustrated in FIG. 5, the image receiver 6 receives the referenceimage and the corresponding image respectively acquired by the firstimage capturing device 12 and the second image capturing device 13 (stepS0). Next, the measurement point receiver 7 receives the measurementpoint input by the user on the reference image through the input unit 4(step S1). Next, the corresponding point searching unit 8 searches forand sets the corresponding point corresponding to the measurement pointreceived by the measurement point receiver 7, on the corresponding image(step S2, corresponding point search step).

Next, the corresponding point searching unit 8 determines whether theposition of the corresponding point on the corresponding image thus setneeds to be adjusted (step S5). In step S5, the corresponding pointsearching unit 8 may determine whether the position of the correspondingpoint needs to be adjusted by automatically determining whether the setposition of the corresponding point corresponds to the measurementpoint. The corresponding point searching unit 8 may also determinewhether the position of the corresponding point needs to be adjusted,with the user determining whether the position of the correspondingpoint set by the corresponding point searching unit 8 corresponds to themeasurement point while referring to the position of the correspondingpoint set by the corresponding point searching unit 8 displayed by thedisplay unit, and inputting a result of the determination through theinput unit 4.

When the corresponding point searching unit 8 determines that thecorresponding point adjustment is required in step S5, the user adjuststhe corresponding point set by the corresponding point searching unit 8through the input unit 4 while referring to the measurement pointperipheral image and the corresponding point peripheral image displayedon the display unit 11 so that the corresponding point furthercorresponds to the position of the measurement point on thecorresponding point peripheral image (corresponding point adjustmentstep). Then, the corresponding point adjusting unit 9 receives theposition of the corresponding point thus adjusted by the user (step S3).Next, the measurement value calculating unit 10 calculates thethree-dimensional coordinates of the measurement point by referring toand based on the camera parameter stored in the storage 3, the positionof the measurement point on the reference image received by themeasurement point receiver 7, and the position of the correspondingpoint on the corresponding image received by the corresponding pointadjusting unit 9 (step S4, calculation step).

When the corresponding point searching unit 8 determines that thecorresponding point does not need to be adjusted in step S5, themeasurement value calculating unit 10 calculates the three-dimensionalcoordinates of the measurement point by referring to and based on thecamera parameter stored in the storage 3, the position of themeasurement point on the reference image received by the measurementpoint receiver 7, and the position of the corresponding point on thecorresponding image set by the corresponding point searching unit 8(step S4′, calculation step).

With the image processing method according to Aspect 3 described above,the corresponding point searching unit 8 determines whether thecorresponding point adjustment by the user is required so that the userdoes not need to unnecessarily perform the corresponding pointadjustment. Thus, the image processing method according to the presentembodiment can be efficiently performed.

(Image Processing Method According to Aspect 4)

FIG. 6 is a flowchart illustrating an image processing method accordingto Aspect 2 of the present embodiment performed by the image processingdevice 1. The image processing method according to the present aspect isthe same as the image processing method according to Aspect 1 exceptthat step S4′ is added and step S6 is further added. Thus, detaildescription or steps denoted with the same reference number as thecounterparts in Aspect 1 will be omitted.

As illustrated in FIG. 6, the image receiver 6 receives the referenceimage and the corresponding image respectively acquired by the firstimage capturing device 12 and the second image capturing device 13 (stepS0), Next, the measurement point receiver 7 receives the measurementpoint input by the user on the reference image through the input unit 4(step S1). Next, the corresponding point searching unit 8 searches forand sets the corresponding point corresponding to the measurement pointreceived by the measurement point receiver 7, on the corresponding image(step S2, corresponding point search step). Next, the measurement valuecalculating unit 10 calculates the three-dimensional coordinates of themeasurement point by referring to and based on the camera parameterstored in the storage 3, the position of the measurement point on thereference image received by the measurement point receiver 7, and theposition of the corresponding point on the corresponding image set bythe corresponding point searching unit 8 (step S4′, calculation step).

Next, the user determines whether the position of the correspondingpoint needs to be adjusted by referring to the three-dimensionalcoordinates of the measurement point calculated by the measurement valuecalculating unit 10 (step S6). Then, the user inputs the result of thisdetermination through the input unit 4 so that the corresponding pointsearching unit 8 can determine whether the position of the correspondingpoint needs to be adjusted. If needed, the user may determine whetherthe position of the corresponding point needs to be adjusted by alsoreferring to the position of the corresponding point set by thecorresponding point searching unit 8. In step S6, whether the positionof the corresponding point needs to be adjusted may be automaticallydetermined with reference to the three-dimensional coordinates of themeasurement point calculated by the measurement value calculating unit10. For example, when the automatic determination is employed, theposition of the corresponding point can be determined to be required ifthe three-dimensional coordinates of the measurement point calculated bythe measurement value calculating unit 10 indicate a position outside ameasurement target range of the stereo camera used for the measurement.

When the measurement value calculating unit 10 determines that thecorresponding point adjustment is required in step S6, the user adjuststhe corresponding point set by the corresponding point searching unit 8by referring to the measurement point peripheral image and thecorresponding point peripheral image displayed on the display unit 11 sothat the corresponding point further corresponds to the position of themeasurement point on the corresponding point peripheral image(corresponding point adjustment step). Then, the corresponding pointadjusting unit 9 receives the position of the corresponding point thusadjusted by the user (step S3). Next, the measurement value calculatingunit 10 calculates the three-dimensional coordinates of the measurementpoint by referring to and based on the camera parameter stored in thestorage 3, the position of the measurement point on the reference imagereceived by the measurement point receiver 7, and the position of thecorresponding point on the corresponding image received by thecorresponding point adjusting unit 9 (step S4, calculation step).

In a case where the measurement value calculating unit 10 determinesthat the corresponding point adjustment is not required in step S6. themeasurement value calculating unit 10 determines that thethree-dimensional coordinates of the measurement point calculated is theactual three-dimensional coordinates of the measurement point in stepS4′, and a series of processes for measuring a measurement value isterminated.

With the image processing method according to Aspect 4 described above,the measurement value calculating unit 10 determines whether thecorresponding point adjustment by the user is required so that the userneeds not to unnecessarily perform the corresponding point adjustment.Thus, the image processing method according to the present embodimentcan be efficiently performed.

(Corresponding Point Adjustment by User)

As described above, in step S3, the user adjusts the position of thecorresponding point on the corresponding image while referring to themeasurement point peripheral image and the corresponding pointperipheral image displayed by the display unit 11. In this section, thisadjustment of the corresponding point by the user as well as themeasurement point peripheral image and the corresponding pointperipheral image used for the adjustment will be described below indetail. FIG. 7 to FIG. 14 referred to in the description belowillustrate images displayed by the display unit 11. In the figures, anupper and lower direction is referred to as a vertical direction and aleft and right direction is referred to as a horizontal direction.

FIG. 7A illustrates a reference image 20, with the measurement point Killustrated in the reference image 20 representing the position of themeasurement point received by the measurement point receiver 7 from theuser. FIG. 7B illustrates an image 21 (measurement point peripheralimage 21) of an area in the periphery of the measurement point K,selected within the reference image 20 by the user, and an image 22(corresponding point peripheral image 22) of an area in the periphery ofa corresponding point found in a corresponding image (not illustrated)by the corresponding point searching unit 8.

As illustrated in FIG. 7B, the display unit 11 displays the measurementpoint peripheral image 21 and the corresponding point peripheral image22 side by side, so that the user can easily visually check whether thecorresponding point searching unit 8 has successfully set thecorresponding point at the measurement point K. With this configuration,the user can adjust the corresponding point through the input unit 4, sothat the corresponding point on the corresponding image corresponds tothe measurement point K on the reference image 20.

The display unit 11 may display the measurement point peripheral imageand the corresponding point peripheral image with the area in theperiphery of the measurement point and the area in the periphery of thecorresponding point in the images enlarged. With this configuration, theuser can adjust the position of the corresponding point through theinput unit 4 while checking positions of the measurement point K and thecorresponding point in detail.

In a case where a result of the search tor the corresponding point bythe corresponding point searching unit 8 is determined to beinappropriate, the user adjusts the corresponding point on the image ofthe periphery of the corresponding point. In a case where the result ofthe search for the corresponding point by the corresponding pointsearching unit 8 is determined to be appropriate, the user does notadjust the corresponding point. The measurement value calculating unit10 calculates the three-dimensional coordinates of the measurement pointfrom the camera parameter of the image captures 2. the position of themeasurement point K on the reference image 20, and the position of thecorresponding point set by the corresponding point searching unit 8. Forexample, the display unit 11 may display options such as “ADJUSTCORRESPONDING POINT” and “PERFORM MEASUREMENT WITHOUT ADJUSTINGCORRESPONDING POINT” so that the measurer 5 can receive a result of theuser determination whether the result of the search for thecorresponding point by the corresponding point searching unit 8 isappropriate. The user can select a step to be performed after the searchfor the corresponding point by the corresponding point searching unit 8by selecting either “ADJUST CORRESPONDING POINT” or “PERFORM MEASUREMENTWITHOUT ADJUSTING CORRESPONDING POINT” by operating a mouse or the likefor the input unit 4. If the display unit 11 is a touch panel alsohaving the function of the input unit 4, the user can select the step tobe performed after the search for the corresponding point by thecorresponding point searching unit 8 by performing a touch operation. Ina case where the corresponding point searching unit 8 searching for thecorresponding point fails to find the corresponding point correspondingto the measurement point K, the corresponding point adjusting unit 9 mayreceive the position of the corresponding point as a result of theadjustment of the position of the corresponding point performed by theuser through the input unit 4 without selecting whether thecorresponding point needs to be adjusted. With this configuration, theuser does not need to determine whether the corresponding point needs tobe adjusted, and thus the image processing method can be efficientlyperformed by the image processing device 1. Alternatively, whether thecorresponding point corresponds to the measurement point K may bedetermined in the following manner. Specifically, the display unit 11may display the measurement result obtained by the measurement valuecalculating unit 10, and the user may determine that the correspondingpoint set by the corresponding point searching unit 8 is inappropriatewhen the displayed measurement result largely differs from an expectedmeasurement result. As in the processes in the flowchart illustrated inFIG. 4 or FIG. 6, the measurement value calculating unit 10 maycalculate the measurement value before the corresponding point isadjusted, the display unit 11 may display the measurement value, thearea in the periphery of the measurement point K, and the area in theperiphery of the corresponding point, and the user may adjust thecorresponding point while referring to these. Alternatively, as in theprocesses in the flowchart illustrated in FIG. 3 or FIG. 4, the user maydirectly proceed to the adjustment of the corresponding point withoutdetermining whether the corresponding point needs to be adjusted.

Next, the process by which the corresponding point is adjusted by thecorresponding point adjusting unit 9 is described. FIG. 8 illustratesthe display unit 11 displaying a reference image 30. The display unit 11illustrated in FIG. 8 is displaying an image 31 (measurement pointperipheral image 31) of an area in the periphery of the measurementpoint K and an image 32 (corresponding point peripheral image 32) of anarea in the periphery of the corresponding point found in thecorresponding image (not illustrated) by the corresponding pointsearching unit 8. When the display unit 11 displays the measurementpoint peripheral image 31 and the corresponding point peripheral image32 at adjacent positions as illustrated in FIG. 8, the user can visuallyrecognize the images at once, and thus can more easily determine whetherthe position of the corresponding point corresponds to the position ofthe measurement point K during the adjustment of the correspondingpoint.

FIG. 9A illustrates an image 40 (measurement point peripheral image 40)of an area in the periphery of the measurement point K and an image 41(corresponding point peripheral image 41) of an area in the periphery ofthe corresponding point according to an aspect of the presentembodiment. The measurement point peripheral image 40 and thecorresponding point peripheral image 41 are displayed to be arrangedside by side along their epipolar lines. The measurement pointperipheral image 40 and the corresponding point peripheral image 41illustrated in FIG. 9A depict a state before the adjustment by the user,and thus the measurement point K does not correspond to thecorresponding point. In each of the measurement point peripheral image40 and the corresponding point peripheral image 41, guides as indicatedby dotted lines are displayed in an overlapping manner. Specifically,the display unit 11 displays a guide (horizontal guide 42) passingthrough the measurement point and the corresponding point and two guides(vertical guides 43 and 44) that are orthogonal to the horizontal guide42, in the measurement point peripheral image 40 and the correspondingpoint peripheral image 41 in an overlapping manner. The vertical guides43 and 44 respectively pass through the measurement point K and thecorresponding point in the images. With this configuration, the user canmore easily notice the displacement between the measurement point K andthe corresponding point, and thus can more easily adjust the position ofthe corresponding point so that the position of the corresponding pointcorresponds to the position of the measurement point K. In this example,the horizontal guide 42 overlaps with the epipolar line of the images ina collinear manner. A point on an image captured by one camera and aposition corresponding to the point on an image captured by the othercamera are positioned on a straight line which is referred to as theepipolar line.

FIG. 9B illustrates the measurement point peripheral image 40 and thecorresponding point peripheral image 41 as a result of the adjustment ofthe corresponding point on the measurement point peripheral image 40 andthe corresponding point peripheral image 41 in FIG. 9A by the user. Itis apparent that the corresponding point corresponds to the measurementpoint K, from comparison between the measurement point peripheral image40 and the corresponding point peripheral image 41 in areas in theperiphery of intersections between the horizontal guide and the twovertical guides. In this manner, the user can adjust the position of thecorresponding point while checking that the image 41 is being adjustedso that the position of the corresponding point corresponds to theposition of the measurement point K, by adjusting the position of thecorresponding point while referring to the guides.

FIG. 10A illustrates art image 50 (measurement point peripheral image50) of an area in the periphery of the measurement point K and an image51 (corresponding point peripheral image 51) of an area in the peripheryof the corresponding point according to an aspect of the presentembodiment. Each of the image 50 and the image 51 includes a subject.This subject has a depth, and thus involves points different from eachother in parallax. Specifically, points in areas in the periphery of themeasurement point K and in the periphery of the corresponding point aredifferent from each other in parallax. In this case, adjustment for thecorresponding image to equalize the parallax for the points in theentire areas displayed is not a feasible option. Thus, the user mayadjust the position of the corresponding point so that a part of thearea in the periphery of the measurement point K corresponds to a partof the area in the periphery of the corresponding point. However, thisonly results in other parts of the images not corresponding to eachother (displacement). All things considered, it is difficult for theuser to perform the adjustment so that the corresponding pointcorresponds to the measurement point K.

FIG. 10B illustrates another mode of the measurement point peripheralimage 50 and the corresponding point peripheral image 51 illustrated inFIG. 100. The display unit 11 displays a horizontal guide 52 andvertical guides 53 and 54 in the measurement point peripheral image 50and the corresponding point peripheral image 51 in an overlappingmanner, as in the measurement point peripheral image 40 and thecorresponding point peripheral image 41 in FIG. 9A and FIG. 9B. The usercan easily perform the adjustment so that the position of thecorresponding point corresponds the position of the measurement point Kby referring to these guides, even when the display areas include areaslargely different from each other in parallax. The corresponding pointmay be adjusted with the user operating the mouse or using the four-waycontroller to input an operation for moving the area in the periphery ofthe corresponding point displayed by the display unit 11. Thecorresponding point is on the epipolar line of the corresponding image,and thus the user may perform the adjustment of the corresponding pointwith the adjustable direction limited to a direction along the epipolarline direction by the corresponding point adjusting unit 9. Thus, therisk of adjusting the corresponding point to a wrong position can bereduced. It is difficult for the user to accurately adjust the positionof the corresponding point along the epipolar line direction through amouse or touch operation. Thus, the corresponding point can be preventedfrom being inappropriately adjusted to be outside the epipolar line withthe corresponding point adjusting unit 9 limiting the user's adjustabledirection. After the adjustment of the corresponding point by the useris completed, the display unit 11 may display a message “TERMINATEADJUSTMENT” for example. Thus, the user can determine whether tocontinue the corresponding point adjustment by selecting whether toterminate the adjustment of the corresponding point through a mouseoperation or the like.

Next, a more preferable display mode of the display unit 11 is describedin detail.

FIG. 11 illustrates the display unit 11 according to one aspect of thepresent embodiment. This display unit 11 is displaying a reference image60 captured by one of two image capturing devices (not illustrated)arranged with the optical axes on the same horizontal plane. The displayunit 11 also displays the measurement point K in the reference image 60in an overlapping manner. The display unit 11 is further displaying animage 61 (measurement point peripheral image 61) or an area, in theperiphery of the measurement point K in the reference image 60 and animage 62 (corresponding point peripheral image 62) of an area in theperiphery of the corresponding point on a corresponding image (notillustrated). These images are displayed side by side along a direction(the vertical direction in this example) orthogonal to the epipolar lineof the images. Specifically, the display unit 11 displays a guide(vertical guide 63) passing through the measurement point K and thecorresponding point and two guides (horizontal guides 64 and 65) thatare orthogonal to the vertical guide 63 in the measurement pointperipheral image 61 and the corresponding point peripheral image 62 inan overlapping manner. The horizontal guides 64 and 65 respectively passthrough the measurement point K and the corresponding point in theimages. The epipolar line of the images captured by the two imagecapturing devices arranged with the optical axes on the same horizontalplane extends along the horizontal direction of the images. Thus, thecorresponding point searching unit 8 only needs to search thecorresponding image along the horizontal direction, that is, theepipolar line direction, and the displacement of the corresponding pointas a result of the search also occurs along the horizontal direction ofthe corresponding image. Thus, the user needs to adjust thecorresponding point only in the horizontal direction of thecorresponding image. As illustrated in FIG. 11, the display unit 11displays the measurement point peripheral image 61 and the correspondingpoint peripheral image 62 side by side along the direction orthogonal tothe epipolar line of the images. Thus, the user can more easilyrecognize the displacement between the epipolar line directions of theimages, compared with the case where the measurement point peripheralimage and the corresponding point peripheral image displayed to bearranged with their epipolar lines arranged in a collinear manner asillustrated in FIG. 9A.

FIG. 12 illustrates the display unit 11 according to one aspect of thepresent embodiment. This display unit 11 is displaying a reference image70 captured by one of two image capturing devices arranged with theoptical axes on the same vertical plane. The display unit 11 isdisplaying the measurement point K in the reference image 70 in anoverlapping manner. The display unit 11 is further displaying an image71 (measurement point peripheral image 71) of an area in the peripheryof the measurement point K and an image 72 (corresponding pointperipheral image 72) of an area in the periphery of the correspondingpoint on a corresponding image. These images are displayed side by sidealong a direction (the horizontal direction in this example) orthogonalto the epipolar line of the images. The display unit 11 displays a guide(horizontal guide 73) passing through the measurement point K and thecorresponding point and two guides (vertical guides 74 and 75) that areorthogonal to the horizontal guide 73 in an overlapping manner in themeasurement point peripheral image 71 and the corresponding pointperipheral image 72 displayed side by side. The vertical guides 74 and475 respectively pass through the measurement point K and thecorresponding point in the images. In a case where the reference imageand the corresponding image are captured by two image capturing devicesarranged side by side with the optical axes on the same vertical plane,the epipolar line of the images extends along the vertical direction ofthe images. The display unit 11 displays the measurement pointperipheral image 71 and the corresponding point peripheral image 72 sideby side along the direction orthogonal to the epipolar line of theimages. The user can adjust the corresponding point while recognizingthe displacement in the images along the epipolar line direction, byreferring to the measurement point peripheral image 71 and thecorresponding point peripheral image 72 displayed side by side.

FIG. 13 illustrates the display unit 11 according to one aspect of thepresent embodiment. This display unit 11 is displaying a reference image80 captured by one of two image capturing devices (not illustrated)arranged with the optical axes on the same horizontal plane as in FIG.11. The display unit 11 is displaying the measurement point K in thereference image 80 in an overlapping manner. The display unit 11 isfurther displaying an image 81 (measurement point peripheral image 81)of an area in the periphery of the measurement point K in the referenceimage 80 and an image 82 (corresponding point peripheral image 82) of anarea in the periphery of the corresponding point on a correspondingimage (not illustrated). These images are displayed side by side along adirection (the vertical direction in this example) orthogonal to theepipolar lines of the images. The display unit 11 displays a guide(vertical guide 83) passing through the measurement point K and thecorresponding, point and two guides (horizontal guides 84 and 85) thatare orthogonal to the vertical guide 83 in an overlapping manner in theimage 81 and the image 82 displayed side by side. The horizontal guides84 and 85 respectively pass through the measurement point K and thecorresponding point in the images. The images 81 and 82 in FIG. 13 andthe images 61 and 62 in FIG. 11 are different from each other in theaspect ratio of the display areas. Specifically, the images 81 and 82 inFIG. 13 have elongated shape extending in the horizontal direction. Theimages 81 and 82 have longitudinal sides extending along the epipolarlines of the images. The corresponding point is on the epipolar line,and thus the user can more easily recognize the displacement between theepipolar line directions by reducing the widths of the measurement pointperipheral image 81 and the corresponding point peripheral image 82 inthe direction orthogonal to the epipolar lines extending along thehorizontal direction of the corresponding image. The display unit 11 maydisplay the measurement point peripheral image 81 and the correspondingpoint peripheral image 82 with elongates shapes (rectangular shapes forexample) extending along the longitudinal direction of the display unit11 as illustrated in FIG. 13, so that the space of the display unit 11can be more effectively used compared with the display mode illustratedin FIG. 11.

FIG. 14 illustrates the display unit 11 according to one aspect of thepresent embodiment. The display unit 11 is displaying a reference image90, a corresponding image 91, an image 94 (measurement point peripheralimage 94) of an area in the periphery of the measurement point ,. and animage 95 (corresponding point peripheral image 95) of an area in theperiphery of the corresponding point. Specifically, the display unit 11displays a guide (vertical guide 96) passing through the measurementpoint K and the corresponding point and two guides (horizontal guides 97and 98) that are orthogonal to the vertical guide 96 in an overlappingmanner in the measurement point peripheral image 94 and thecorresponding point peripheral image 95 displayed side by side. Thehorizontal guides 97 and 98 respectively pass through the measurementpoint K and the corresponding point in the images. The display unit 11is displaying a marker 92 in the reference image 90 in an overlappingmanner. The marker 92 indicates the area in the periphery of themeasurement point K displayed in the image 94. The display unit 11 isdisplaying a marker 93 in the corresponding image 91 in an overlappingmanner. The marker 93 indicates the area in the periphery of thecorresponding point displayed in the image 95. With the positions wherethe reference image 90 and the corresponding image 91 are clipped todisplay the measurement point peripheral image 94 and the correspondingpoint peripheral image 95 thus displayed on the reference image 90 andon the corresponding image 91, the user can be prevented from adjustingthe corresponding point to be at the position not corresponding to themeasurement point K. A plurality of similar areas are found as a resultor comparison using clipped areas only, especially when the measurementtarget is a subject with repetitive similar patterns. As a result, theuser might determine an inappropriate position to be the positioncorresponding to the position of the measurement point K. In view ofthis, the clipped display areas are displayed on the reference image 90and the corresponding image 91, so that the user can recognize theclipped positions of the display images. Thus, the corresponding pointcan be more easily adjusted to correspond to the measurement point Keven in a case where the measurement target is the subject withrepetitive similar patterns.

The display unit 11 may display the measurement point peripheral image94 and the corresponding point peripheral image 95 that are monochromeimages. For example, when the reference image 90 and the correspondingimage 91 are RGB color images, the display unit 11 may display themeasurement point peripheral image 94 and the corresponding pointperipheral image 95 of the color images to be monochrome images by usingthe G channel only, to be free of a color registration error that mayoccur with the color images. Thus, the user can easily adjust thecorresponding point. The display unit 11 may calculate a Y value of theRGB values of the reference image 90 and the corresponding image 91, andmay display the measurement point peripheral image 94 and thecorresponding point peripheral image 95 as monochrome images using the Yvalue only. The display unit 11 may display the corresponding imagesearched by the corresponding point searching unit 8 for thecorresponding point, as a corresponding point adjustment image for theuser.

The display unit 11 may display the image of the area in the peripheryof the measurement point and an image of the area in the periphery ofthe corresponding point in an overlapped manner, and may display theoverlapped images as the corresponding point adjustment image for theuser. With this configuration, the user can recognize a state where thecorresponding point is displaced from the measurement point by findingdoubled edges or the like on the two overlapped images due to thecorresponding point failing to correspond to the measurement point. Thedisplay unit 11 may display two monochrome images with different colorsin an overlapping manner. With this configuration, the user can easilyrecognize the state where the corresponding point is displaced. Forexample, the display unit 11 may display an image of an area in theperiphery of the measurement point with a color G and an image of anarea in the periphery of the corresponding point with a color R. As aresult, a position where the corresponding point and the measurementpoint are appropriately overlapped with each other is colored in yellow(mixture of the colors R and G. At a position where the position of thecorresponding point is displaced from the position of the measurementpoint, the corresponding point has one of components R and G beinglarger than the other to be displayed as R or a G edge. Thus, the usercan more easily recognize the displacement between the measurement pointand the corresponding point.

The image processing device 1 according to the present embodiment may beconfigured to be capable of changing the magnifications of themeasurement point peripheral image and the corresponding pointperipheral image displayed by the display unit 11 for the adjustment ofthe corresponding point by the user. When the display unit 11 displaysthe measurement point peripheral image and the corresponding pointperipheral image in the actual scale, the adjustment accuracy of thecorresponding point is in a unit of a pixel, and thus the user canperform the adjustment with an accuracy in the unit of a pixel. If thedisplay unit 11 has small pixels, it is difficult for the user tovisually recognize the displacement with the accuracy in the unit of apixel. Thus, the display unit 11 displays the measurement pointperipheral image and the corresponding point peripheral image enlargedto be larger than the actual scale, so that the user can easilyrecognize the displacement between the enlarged images. For example, thedisplay unit 11 may display “ZOOM IN” and “ZOOM OUT” so that the usercan increase or reduce the display size by selecting any one of thesethrough a mouse operation or the like. In a case where the display sizeis increased, the adjustment can be further performed with an accuracyin a unit of a sub pixel smaller than a pixel. The display unit 11 maychange the display magnification of the measurement point peripheralimage for adjusting the corresponding point and the corresponding pointperipheral image in accordance with a parallax value. In thestereoscopic measurement, a longer distance between the two imagecapturing devices and the measurement point leads to a largermeasurement error. Thus, the parallax needs to be accurately calculatedto accurately measure the subject far from the two image capturingdevices. Thus, in a case where the distance between the two imagecapturing devices and the measurement point is long, and thus theparallax is small, the display unit 11 displays the measurement pointperipheral image for adjusting the corresponding point and thecorresponding point peripheral image with a large magnitude. Thus, theuser can adjust the corresponding point position in detail by referringto the measurement point peripheral image and the corresponding pointperipheral image enlarged, whereby the measurement point can be measuredwith a higher accuracy. A long distance between the two image capturingdevices and the subject results in an image captured with a smallersubject compared with that captured with the distance being short. Thus,the subject at a position far from the image capturing devices aredisplayed with an increased magnification so that the user can moreeasily recognize the displacement.

In the present embodiment described above, the image of the area in theperiphery of the measurement point in the reference image and the imageof the area in the periphery or the corresponding point in thecorresponding image are displayed side by side. The user performs theadjustment by referring to the measurement point peripheral image andthe corresponding point peripheral image displayed side by side, so thatthe position of the corresponding point corresponds to the position ofthe measurement point. With the image of the area in the periphery ofthe measurement point in the reference image and the image of the areain the periphery of the corresponding point in the corresponding imagedisplayed side by side, the user can adjust the corresponding point,while associating the corresponding image and the reference image witheach other. The user refers to the position of the corresponding pointaccurately adjusted, and thus the measurement point can be measured witha higher accuracy.

Second Embodiment

Next, a second embodiment of the present invention will be described. Inthe second embodiment, the user that performs stereoscopic measurementby using images captured with three or more image capturing devicesadjusts a corresponding point by using a corresponding point adjustmentmethod enabling the position of the corresponding point to he easily andaccurately adjusted. Thus, measurement accuracy is improved. In a casewhere a plurality of corresponding images are captured by a plurality ofimage capturing devices as in the present embodiment, the imageprocessing method according to the first embodiment may he performed fora combination between the reference image and each of the correspondingimages. Thus, the parallaxes of the measurement point in the number thatis equal to the number of such combinations are obtained. The accuracyof the three-dimensional coordinates of the measurement point finallyobtained can be improved by taking an average or the like of theparallaxes of the measurement point.

An image processing device (not illustrated) according to the presentembodiment has the same configuration as the image processing device 1according to the first embodiment, except that the image capturer 2includes three or more image capturing devices. Thus, the description onthe members already described in the first embodiment will be omitted.The image processing method according to the present embodiment is thesame as the one described above except for display modes of themeasurement point peripheral image and the corresponding pointperipheral image displayed by the display unit 11 in accordance with thesteps in the first embodiment. Thus, in the description below, only thedisplay modes of the measurement point peripheral image and thecorresponding point peripheral image displayed by the display unit 11will be described in detail.

FIG. 15A illustrates images 100, 101, and 102 captured by three imagecapturing devices arranged in an L shape along two directions includingthe horizontal direction (first direction) and a vertical direction(second direction). The image 101 serves as a reference image. The image102 is a corresponding image captured by the image capturing devicearranged next to the image capturing device that has captured the image101 serving as the reference image, in the horizontal direction. Theimage 100 is a corresponding image captured by the image capturingdevice arranged next to the image capturing device that has captured theimage 101 serving as the reference image, in the vertical direction.Each of the images includes a subject. The reference image 101 includesthe measurement point K set by the user. The stereoscopic measurementmay be performed for the measurement point K on the reference image 101,by using the reference image 101 and the image 102 arranged side by sidein the horizontal direction on the images, or by using the referenceimage 101 and the image 100 arranged side by side in the verticaldirection on the images. The three-dimensional coordinates of themeasurement point as viewed from the image capturing device that hascaptured the reference image 101 is the same between these cases. Thus,d₂−d₁×(B₂/B₁) . . . (4) holds true where B₁ represents the baselinelength between the image capturing devices arranged side by side in thehorizontal direction, B₂ represents the baseline length between theimage capturing devices arranged side by side in the vertical direction,Z₁ represents a Z coordinate obtained from the reference image 101 andthe corresponding image 102 (a coordinate, in coordinates based on theimage capturing device that has captured the reference image 101, in adirection from the image capturing device to the subject), Z₂ representsa Z coordinate obtained from the reference image 101 and thecorresponding image 100, d₁ represents a parallax calculated from themeasurement point K on the reference image 101 and the correspondingpoint on the corresponding image 102, d₂ represents a parallax of themeasurement point calculated from the measurement point K on thereference image 101 and the corresponding point on the correspondingimage 100, and the three-dimensional coordinates of the measurementpoint calculated are the same so that Z₁=Z₂ holds true. With therelationship between d₁ and d₂ in Formula (4) satisfied, thecorresponding points on the two corresponding images may be adjusted inthe stereoscopic measurement using images captured with three imagecapturing devices. In such a case, the corresponding points on onecorresponding image 102 and the other corresponding image 100 may besimultaneously adjusted in an interlocking manner, so that thecorresponding points correspond to the measurement point K on thereference image 101. When the corresponding point on the onecorresponding image 102 is adjusted to be at a position where a parallaxd1 is obtained relative to the measurement point K on the referenceimage 101, the parallax d₂ between the measurement K on the referenceimage 101 and the corresponding point on the other corresponding image100 may be adjusted to be a value in Formula (4), so that thecorresponding points on the two corresponding images can besimultaneously adjusted. With this configuration, the load on the usercan be reduced compared with a configuration where the userindependently performs each of the adjustment of the corresponding pointwith reference to the reference image 101 and the one correspondingimage 102 and the adjustment of the corresponding point with referenceto the reference image 101 and the other corresponding image 102. Withthe corresponding points simultaneously adjusted, the correspondingpoints that are difficult to adjust with reference to the onecorresponding image 102 can be adjusted with reference to the othercorresponding image 100 with which the adjustment can be easilyperformed. Thus, the user can accurately adjust the corresponding point.

FIG. 15B illustrates a measurement point peripheral image and acorresponding point peripheral image for adjusting a corresponding pointcorresponding to the measurement point K. An image 103 and an image 104are respectively an image of an area in the periphery of the measurementpoint K on the reference image 101 and an image of an area in theperiphery of the corresponding point on the corresponding image 100. Animage 108 and an image 109 are respectively an image or an area in theperiphery of the measurement point K on the reference image 101 and animage of an area in the periphery of the corresponding point on thecorresponding image 102. The display unit 11 displays the image 103 andthe image 104 side by side and displays the image 108 and the image 109side by side so that the user can easily adjust the correspondingpoints. Specifically, the display unit 11 displays a guide (horizontalguide 105) passing through the measurement point K and the correspondingpoint and two guides (vertical guides 106 and 107) that are orthogonalto the horizontal guide 105 in an overlapping manner in the image 103and the image 104 displayed side by side. The vertical guides 106 and107 respectively pass through the measurement point K and thecorresponding point in the images. Specifically, the display unit 11displays a guide (vertical guide 110) passing through the measurementpoint K and the corresponding point and two guides (horizontal guides111 and 111) that are orthogonal to the vertical guide 110 in anoverlapping manner in the image 108 and the image 109 displayed side byside. The horizontal guides 111 and 111 respectively pass through themeasurement point K and the corresponding point in the images.

The image 103 that is a measurement point peripheral image of thereference image 101 and the image 104 that is a corresponding pointperipheral image of the corresponding image 100 involve a parallax inthe vertical direction on the image 103 and the image 104. Thus, theuser adjusts the corresponding point in the vertical direction on theimage 104. The image 108 that is a measurement point peripheral image ofthe reference image and the image 109 that is a corresponding pointperipheral image of the corresponding image 102 involve a parallax inthe vertical direction on the image 108 and the image 109. Thus, theuser adjusts the corresponding point in the vertical direction on theimage 109.

Next, a display mode of a measurement point periphery image and acorresponding point periphery image displayed by the display unit 11according to another aspect of the image processing method of thepresent embodiment is described.

FIG. 16 illustrates a measurement point peripheral image and acorresponding point peripheral image in a case where stereoscopicmeasurement is performed with images captured by three image capturingdevices arranged to be in an L shape. An image 121 is an image(measurement point peripheral image 121) illustrating an area in theperiphery of the measurement point K on the reference image 101, and animage 121 is an image (corresponding point peripheral image 120)illustrating an area in the periphery of the corresponding point on thecorresponding image 100. The measurement point peripheral image 121 andthe corresponding point peripheral image 120 are arranged side by sidein the vertical direction on the images. Similarly, an image 125 is animage (measurement point peripheral image 125) illustrating an area inthe periphery of the measurement point K on the reference image 101, andan image 126 is an image (corresponding point peripheral image 126)illustrating an area in the periphery of the corresponding point on thecorresponding image 102. The measurement point peripheral image 125 andthe corresponding point peripheral image 126 are arranged side by sidein the vertical direction on the images. The display unit 11 displays aguide (vertical guide 122) passing through the measurement point K andthe corresponding point and two guides (horizontal guides 123 and 124)that are orthogonal to the vertical guide 122 in an overlapping mannerin the images 120 and 121 displayed side by side. The horizontal guides123 and 124 respectively pass through the measurement point K and thecorresponding point in the images. The display unit 11 displays a guide(vertical guide 122 described above) passing through the measurementpoint K and the corresponding point and two guides (horizontal guides127 and 128) that are orthogonal to the vertical guide 122 in anoverlapping manner in the images 125 and 126 displayed side by side. Thehorizontal guides 127 and 128 respectively pass through the measurementpoint K and the corresponding point in the images.

The images 121, 120, 125, and 126 respectively correspond to the images103, 104, 108, and 109 in FIG. 15B, and the images 121 and 120 areobtained by rotating the respective images 103 and 104 by 90 degrees.The images 121 and 120 as a result of rotating the images arranged sideby side in the vertical direction on the images by 90 degrees arearranged side by side in the horizontal direction on the images to bearranged in manner similar to that of the images 103 and 104 in FIG.15B. Thus, the corresponding point can be adjusted in a manner similarto that in the adjustment of the corresponding point in the measurementpoint periphery image and the corresponding point periphery image inFIG. 15B.

When the corresponding point adjusting unit 9 adjusts the correspondingpoint based on an input from the user through a mouse operation orfour-way controller input, a more natural operation can be achieved witha direction or the operation matching the direction of the movement ofthe corresponding point on, the image. An adjustment direction may beset to be the same for the corresponding points on the images 120 and126, which are respectively parts of the corresponding image 100 and thecorresponding image 102, and the corresponding points on the images 120and 126 may be interlocked as described above. With this configuration,the user can check the position of the one of the corresponding pointsinterlocked in a single direction while adjusting the other one of thecorresponding points. Thus, the corresponding points can be moreaccurately adjusted.

An image may be rotated by 90 degrees to be displayed with the epipolarhue direction of the image rotated by 90 degrees. Thus, arrangementdirections of a measurement point peripheral image for adjusting acorresponding point and a corresponding point peripheral image by 90degrees. FIG. 16 illustrates the measurement point peripheral image 121and the corresponding point peripheral image 120 as a result of rotatingimages arranged side by side in the horizontal direction of the imagesso that the images are arranged side by side in the vertical directionof the display unit 11. As a result, the side by side arrangement in adirection orthogonal to the epipolar line direction can be maintained.The user can adjust the corresponding points on images captured by fouror more image capturing devices in a similar manner. Specifically, thecorresponding point adjusting unit 9 may adjust the positions of thecorresponding. points OD the corresponding images in an interlockingmanner so that the positions of the corresponding points correspond tothat of the measurement point on the reference image. Thus, the user caneasily adjust the corresponding points. The measurement point peripheralimage and the corresponding point peripheral image for adjusting acorresponding point may be rotated so that the direction of theadjustment by the user becomes the same between the correspondingimages. Thus, the user can even more easily adjust the correspondingpoints.

Third Embodiment (Embodiment Implemented by Software)

Functional blocks or the image processing device 1 described above maybe implemented with a computer. This configuration may be realized byrecording a program (image processing program) for realizing the imageprocessing device 1 on a computer-readable recording medium and causinga computer system to read the program recorded on the recording mediumfor execution. The “computer-readable recording medium” refers to aportable medium such as a flexible disk, a magneto-optical disk, a ROM,and a CD-ROM, and a storage device such as a hard disk built into thecomputer system. Moreover, the “computer-readable recording medullar”may include a medium that dynamically retains the program for a shortperiod of time, such as a communication line that is used to transmitthe program over a network such as the Internet or, over a communicationline such as a telephone line, and a medium that retains, in that case,the program for a fixed period of time, such as a volatile memory withinthe computer system which functions as a server or a client.Furthermore, the program may be configured to realize some of thefunctions described above, and also may be configured to be capable ofrealizing the functions described above in combination with a programalready recorded in the computer system.

Additionally, the image processing method according to the embodimentsdescribed above may be partially or completely realized as a Large ScaleIntegration (LSI) circuit, which is a typical integrated circuit. Thefunctional blocks for the stereoscopic measurement may be individuallyrealized as chips, or may he partially or completely integrated into achip. The circuit integration technique is not limited to LSI, and theintegrated circuits for the functional blocks may be realized asdedicated circuits or a multi-purpose processor. Furthermore, in a casewhere with advances in semiconductor technology, a circuit integrationtechnology with which an LSI is replaced appears, it is also possible touse an integrated circuit based on the technology.

The embodiment of the present invention has been described in detailabove referring, to the drawings, but the specific configuration is notlimited to the above embodiments and various amendments can be made to adesign that fall within the scope that does not depart from the gist ofthe present invention.

Supplement

An image processing device (1) according to Aspect 1 of the presentinvention is an image processing device that calculatesthree-dimensional coordinates of a measurement point on a subject basedon a reference image of the subject captured by a first image capturingdevice and a corresponding image of the subject captured by a secondimage capturing device, and includes a corresponding point searchingunit (8) that searches the corresponding image for a corresponding pointcorresponding to the measurement point on the reference image, and adisplay unit (11) that displays a measurement point peripheral imagethat is an image of an area in a periphery of the measurement pointextracted from the reference image and a corresponding point peripheralimage that is an image of an area in a periphery of the correspondingpoint extracted from the corresponding image, the measurement pointperipheral image and the corresponding point peripheral image beingdisplayed side by side, to adjust a position of the corresponding pointfound by the corresponding point searching unit.

With the configuration described above, the image of the area in theperiphery of the measurement point in the reference image and the imageof the area in the periphery of the corresponding point in thecorresponding image are displayed side by side. The user performs theadjustment by referring to the measurement point peripheral image andthe corresponding point peripheral image displayed side by side, so thatthe position of the corresponding point corresponds to the position ofthe measurement point. With the image of the area in the periphery ofthe measurement point in the reference image and the image of the areain the periphery of the corresponding point in the corresponding imagedisplayed side by side, the user can adjust the corresponding pointwhile associating the corresponding image and the reference image witheach other. The user refers to the position of the corresponding pointaccurately adjusted, and thus the measurement point can be measured witha higher accuracy.

With the image processing device (1) according to Aspect 2 of thepresent invention, in Aspect 1 described above, the measurement pointperipheral image and the corresponding point peripheral image aredisplayed side by side in a direction orthogonal to an epipolar line ofthe reference image and the corresponding image.

With the configuration described above, the user can easily check thedisplacement in the images in a direction of the epipolar line, and thusthe corresponding point can be more accurately adjusted.

With the image processing device (1) according to Aspect 3 of thepresent invention, in Aspect 1 or 2 described above, the correspondingpoint peripheral image is formed in a rectangular shape having longsides extending along an epipolar line of the reference image and thecorresponding image and short sides extending along a directionorthogonal to the epipolar line.

With the configuration described above, the user can more easily adjustthe corresponding point along the epipolar line.

With the image processing device (1) according to Aspect 4 of thepresent invention, in any one of Aspects 1 to 3 described above, thedisplay unit (11) displays a guide for adjusting the position of thecorresponding point to be overlapped with the measurement pointperipheral image and the corresponding point peripheral image.

With the configuration described above, the user can adjust thecorresponding point while checking the guide, whereby the correspondingpoint can be more accurately adjusted.

The image processing device (1) according to Aspect 5 of the presentinvention, in any one of Aspects 1 to 4 described above, furtherincludes an input unit (4) to which an instruction related to movementof the corresponding point is input, and a corresponding point adjustingunit (9) that adjusts the position of the corresponding point based onthe instruction input to the input unit. The corresponding pointadjusting unit adjusts the position of the corresponding point only in adirection along an epipolar line of the reference image and thecorresponding image.

With the configuration described above, the adjustable direction by theuser is limited so that the corresponding point can be prevented frombeing inappropriately adjusted to be outside the epipolar line.

The image processing device (1) according to Aspect 6 of the presentinvention, in Aspect 5 described above, calculates the three-dimensionalcoordinates based on another corresponding image of the subject capturedby a third image capturing device, the reference image, and thecorresponding image.

the corresponding point searching unit (8) searches the othercorresponding image for another corresponding point corresponding to themeasurement point, and

the corresponding point adjusting unit (9) adjusts a position of theother corresponding point in association with the adjustment for theposition of the corresponding point.

With the configuration described above, the corresponding point on onecorresponding image adjusted by the user is interlocked with one or morecorresponding point on another corresponding image. Thus, the user cansimultaneously adjust a plurality of corresponding points. With thisconfiguration, the load on the user can be reduced compared with aconfiguration where the user independently performs the adjustment withreference to the reference image and the corresponding image for anumber of times that is the same as the number of possible combinationsbetween the reference images and corresponding images. With thecorresponding points simultaneously adjusted, the corresponding pointsthat are difficult to adjust with reference to the one correspondingimage can be adjusted with reference to the other corresponding imagewith which the adjustment can be easily performed. Thus, the user canaccurately adjust the corresponding point.

With the image processing device (1) according to Aspect 7 of thepresent invention, in Aspect 6 described above,

the first and the second image capturing devices are arranged along afirst direction,

the first and the third image capturing devices are arranged along asecond direction intersecting with the first direction,

another corresponding point peripheral image that is an image of an areain a periphery of the other corresponding point extracted from the othercorresponding image, the measurement point peripheral image, and thecorresponding point peripheral image are displayed with an adjustmentdirection of the position of the other corresponding point in the othercorresponding point peripheral image being same as an adjustmentdirection of the position of the corresponding point in thecorresponding point peripheral image.

With the configuration described above, the user can check the positionof one of the corresponding points interlocked in the same direction,when adjusting the other one of the corresponding points. Thus, thecorresponding point can be adjusted more accurately.

An image processing method according to Aspect 8 of the presentinvention is an image processing method for calculatingthree-dimensional coordinates of a measurement point on a subject basedon a reference image of the subject captured by a first image capturingdevice and a corresponding image of the subject captured by a secondimage capturing device, and includes searching the corresponding imagefor a corresponding point corresponding to the measurement point on thereference image, and displaying a measurement point peripheral imagethat is an image of an area in a periphery of the measurement pointextracted from the reference image and a corresponding point peripheralimage that is an image of an area in a periphery of the correspondingpoint extracted from the corresponding image, the measurement pointperipheral image and the corresponding point peripheral image beingdisplayed side by side, to adjust a position of the corresponding pointfound in the searching.

With the configuration described above, an effect is exerted similarlyto the image processing device according to Aspect 1 described above.

The present invention is not limited to each of the above-describedembodiments. It is possible to make various modifications within thescope of the claims. An embodiment obtained by appropriately combiningtechnical elements each disclosed in different embodiments falls alsowithin the technical scope of the present invention Further, whentechnical elements disclosed in the respective embodiments are combined,it is possible to form a new technical feature.

CROSS-REFERENCE OF RELATED APPLICATION

This application claims the benefit of priority to JP 2016-107767 filedon May 30, 2016, which is incorporated herein by reference in itsentirety.

REFERENCE SIGNS LIST

1 image processing device

2 Image capturer

3 Storage

4 Input unit

5 Measurer (calculating unit)

6 Image receiver

7 Measurement point receiver

8 Corresponding point searching unit.

9 Corresponding point adjusting unit

10 Measurement value calculating unit

11 Display unit

12 First image capturing device

13 Second image capturing device

14, 16 Image sensor

15, 17 Lens

1. (canceled)
 2. An image processing device comprising: a correspondingpoint searching unit configured to search a corresponding image for acorresponding point corresponding to a measurement point on a referenceimage obtained by capturing an image of a subject, the correspondingimage being obtained by capturing an image of the subject from a pointof view different from a point of view for the reference image; acorresponding point adjusting unit that causes a display unit to displaya measurement point peripheral image that is an image of an area in aperiphery of the measurement point extracted from the reference imageand a corresponding point peripheral image that is an image of an areain a periphery of the corresponding point extracted from thecorresponding image, the measurement point peripheral image and thecorresponding point peripheral image being displayed side by side in adirection orthogonal to an epipolar line of the reference image and thecorresponding image, and is configured to adjust a position of thecorresponding point based on an instruction input to an input unit; anda calculating unit configured to calculate three-dimensional coordinatesof the measurement point on the subject based on a position of themeasurement point on the reference image and the position of thecorresponding point on the corresponding image.
 3. The image processingdevice according to claim 2, wherein the corresponding point peripheralimage is formed in a rectangular shape having long sides extending alongan epipolar line of the reference image and the corresponding image andshort sides extending along a direction orthogonal to the epipolar line.4. The image processing device according to claim 2, wherein thecorresponding point adjusting unit causes the display unit to displaythe measurement point to be overlapped with the measurement pointperipheral image and to display a corresponding point found by thecorresponding point searching unit to be overlapped with thecorresponding point peripheral image.
 5. The image processing deviceaccording to claim 2, wherein the corresponding point adjusting unitcauses the display unit to display a guide for adjusting the position ofthe corresponding point to be overlapped with the measurement pointperipheral image and the corresponding point peripheral image.
 6. Theimage processing device according to claim 2, wherein the correspondingpoint adjusting unit adjusts the position of the corresponding pointonly in a direction along an epipolar line of the reference image andthe corresponding image.
 7. The image processing device according toclaim 5 further comprising: a first image capturing device configured togenerate the reference image; and a second image capturing deviceconfigured to generate the corresponding image, wherein the imageprocessing device calculates the three-dimensional coordinates based onanother corresponding image of the subject captured by a third imagecapturing device, the reference image, and the corresponding image, thecorresponding point searching unit searches the other correspondingimage for another corresponding point corresponding to the measurementpoint, and the corresponding point adjusting unit adjusts a position ofthe other corresponding point in association with an adjustment for theposition of the corresponding point.
 8. The image processing deviceaccording to claim 7, wherein the first and the second image capturingdevices are arranged along a first direction, the first and the thirdimage capturing devices are arranged along a second directionintersecting with the first direction, and another corresponding pointperipheral image that is an image of an area in a periphery of the othercorresponding point extracted from the other corresponding image, themeasurement point peripheral image, and the corresponding pointperipheral image are displayed with an adjustment direction of theposition of the other corresponding point in the other correspondingpoint peripheral image being the same as an adjustment direction of theposition of the corresponding point in the corresponding pointperipheral image.
 9. The image processing device according to claim 2further comprising the display unit.
 10. An image processing methodcomprising: searching a corresponding image for a corresponding pointcorresponding to a measurement point on a reference image obtained bycapturing an image of a subject, the corresponding image being obtainedby capturing an image of the subject from a point of view different froma point of view for the reference image; causing a display unit todisplay a measurement point peripheral image that is an image of an areain a periphery of the measurement point extracted from the referenceimage and a corresponding point peripheral image that is an image of anarea in a periphery of the corresponding point extracted from thecorresponding image, the measurement point peripheral image and thecorresponding point peripheral image being displayed side by side in adirection orthogonal to an epipolar line of the reference image and thecorresponding image, and adjusting a position of the corresponding pointbased on an instruction input to an input unit; and calculatingthree-dimensional coordinates of the measurement point on the subjectbased on a position of the measurement point on the reference image andthe position of the corresponding point on the corresponding image. 11.An image processing program causing a computer function as the imageprocessing device according to claim 2, wherein the image processingprogram causes the computer to function as the corresponding pointsearching unit, the corresponding point adjusting unit, and thecalculating unit.